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Information about AEL projects, programs, and services is available by writing or calling 
AEL. 
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ABSTRACT 



Education reform and technical assistance systems have been criticized for their inability 
to sustain significant and long-term improvements in teaching and learning. One method 
that has shown success has been the use of an external facilitator in comprehensive 
school change efforts. An external facilitator aims to help schools and districts 
internalize the practices and processes that result in continuous improvement. In a 
collaboration with the Virginia Department of Education, AEL conducted several diverse 
technical assistance activities designed to increase the capacity of external facilitators 
working in the Governor’s Best Practice Centers to effectively work with school and 
school divisions (districts). Four activities are described in this report, including (1) the 
facilitation of strategic planning for the Best Practice Centers, (2) a workshop on data- 
based decision making, (3) an algebra study group to finalize and test elements within a 
proposed algebra resource compendium, and (4) a training for external facilitators 
working with Comprehensive School Reform Demonstration program. 

It may be that as AEL works in partnership with others to build capacity for state- 
directed technical assistance for education improvement initiatives, greater flexibility of 
each partner is required to work directly with schools and districts. The success of AEL’s 
work with the Algebra I study group in which AEL staff were able to work directly with 
individual teachers through collaborative efforts and assistance of the GBPC in the region 
suggests this. Obviously, the effectiveness with which AEL carried out its work with the 
GBPCs may have been influenced by the political realities and uncertainties that they 
confronted as new creations of the state department of education. As a viable partner 
seeking to serve as a catalyst for stimulating opportunities for schools and divisions to 
access needed “external” technical assistance for school improvement initiatives, AEL 
will need to continuously build and nurture short- and long-term collaborative 
relationships when the development of products involves several groups who may have 
competing priorities. 
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Supporting Education Organizations for School Improvement 

Education reform and technical assistance systems have been criticized for their 
inability to support significant and sustainable improvements in teaching and learning, 
despite the investment of substantial public funds. Some of the factors that often defeat 
the creation of meaningful change are short-term, stop-gap professional development 
workshops; misfits between innovations and school contexts; and the challenges of 
adapting reform materials developed elsewhere. Moreover, schools and districts often 
lack what may be called an external facilitator. The external facilitator function, carried 
out by either an individual or team, assists schools in developing their capacity for 
continuous improvement. AEL envisions the external facilitator as responsible for 
analyzing the reform-minded school from an objective vantage point and guiding the 
school through the reform process. 

In its 2000 Updated Annual Plan (1999), AEL defines the external facilitator role 
to include a variety of important tasks and foci, including 

1. focusing on whole-school improvement and ensuring that all strategies fit into a 
comprehensive framework 

2. promoting a culture of informed decision making driven by R&D-based 
knowledge and action research 

3. monitoring the process of school reform (with key school personnel) so that 
problems can be identified and addressed expediently 

4. establishing a support infrastructure at the school for change to occur 

5. involving school staff in ongoing monitoring and evaluation of the 
implementation, viewed in conjunction with student performance 

6. providing school information and direction to external technical assistance 
providers to ensure that products and services are appropriate to the school 
context. 

AEL contends that the assistance of external facilitators is critical to the success 
of technical assistance and school change. Ultimately, however, the external facilitator 
aims to help schools and districts internalize practices and processes that result in 
continuous improvement. 

In response to the challenges of supporting school reform, AEL has proposed a 
new approach to technical assistance based on action research, quick response to requests 
for aid, and collaborative product and process development. A multi-pronged strategy, 
this innovative form of technical assistance has included intensive site work and the 
development of a curriculum mapping and alignment process. Permeating this work has 
been the external facilitator function. These efforts have taken shape under Task 6 of the 
AEL 2000 Updated Annual Plan. 
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Governor’s Best Practice Centers 

As part of its new approach to technical assistance, AEL proposed to support 
other selected education organizations in incorporating the external facilitator function 
into their services. The first of these efforts has been to assist the newly organized 
network of Governor’s Best Practice Centers (GBPC) in Virginia. Established in 
September 1998, the GBPCs were designed to work with local school divisions in 
meeting the new Virginia Standards of Accreditation through successful student 
achievement of the Standards of Learning (SOL) as measured by the SOL tests. Three 
centers were established in September 1998, and five more in September 1999. Each 
GBPC serves a Superintendents’ Regional Study Group, or educational region, consisting 
of approximately 15 to 20 school divisions, and about 250 schools (Virginia Department 
of Education, 2000). 

The Virginia Department of Education (VDOE) requested AEL’s support with the 
training of GBPC staff and developing products and services tailored to specific school 
division, regional, or statewide needs. To this end, AEL developed a research and 
development project to help build the capacity of the GBPCs to in turn build the capacity 
of schools for meeting the SOLs. Activities of this project have included 

• assistance with GBPC strategic planning 

• training on assessment and data-based decision making in the development site of 
Scott County, and the attendant development of a handbook for use by GBPC 
staff 

• the development and refinement, through the facilitation of a teacher study group, 
of a resource compendium for assisting Algebra I teachers to help their students 
achieve the SOLs and 

• training of external facilitators assigned to work with schools receiving 
Comprehensive School Reform Demonstration (CSRD) program funds. 

The purpose of this report is to document work undertaken through Task 6, 
Activity 6.1.2. In addition, this report is intended to present evaluation findings about 
these efforts so that AEL staff may make any needed modifications in their approach 
should they continue to support intermediary education organizations. 

The primary audience for this narrative includes AEL staff responsible for 
developing and implementing Task 6 work. Other audiences are the U.S. Department of 
Education’s Office of Educational Research and Improvement (OERI), from whom AEL 
receives funding; other researchers and practitioners interested in supporting intermediary 
education entities; and participants in the work described in this report. 
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Findings 

Descriptive and evaluation findings are presented in this section of the report. 

Each of the four major efforts undertaken as part of Task 6 is described below. 

Strategic Planning 

To support the new GBPCs, AEL’s Resident Director for Virginia attended 
VDOE meetings at which GBPC issues were discussed. During these meetings, the 
Resident Director established relationships with state department staff leading the GBPC 
initiative and garnered their support for AEL’s new research and development project on 
assistance to intermediary organizations charged with the external facilitator role. 
Collaboration between AEL and the GBPC network would both inform AEL’s 
knowledge base about the external facilitator function and assist the network in building 
its ability to serve schools and divisions in that capacity. 

As a result of connections with VDOE and GBPC personnel, two AEL staff 
members assisted GBPC staff in preparing strategic planning retreats on February 18, 
1999 and February 24-25, 1999, in Richmond, Virginia. An AEL staff member facilitated 
the February 18 session, and AEL’s Resident Director served as session recorder during 
the second and third strategic planning sessions. 

During the strategic planning process, VDOE and GBPC staff developed and 
refined their guiding belief, mission, goal, and objectives statements. GBPC staff also 
clarified their definition of “best practices” such that various levels of best practice were 
identified based upon certain types of evaluative information. GBPC staff brainstormed 
the kinds of assistance activities that should be provided by the centers to schools and 
divisions under the categories of curriculum alignment, instruction, assessment, 
leadership, and technology. Participants in the planning sessions additionally identified a 
variety of issues schools must confront and strategies they might undertake in order to 
improve student achievement. Priority activities were decided for each regional GBPC as 
well. 



Ultimately, AEL’s participation in the strategic planning process with the 
directors of the initial three GBPCs provided essential information and direction for the 
five additional GBPCs. A total of eight GBPCs now serve school divisions in the state. In 
addition, GBPC staff agreed to collaborate with AEL in the development and production 
of an assessment handbook and an Algebra I resource guide. 

Scott County Strategies for Improving Instruction Workshop 

As part of an initiative to provide school divisions with data-based decision 
making skills in order to increase student achievement on the Virginia SOLs, AEL 
developed a handbook on using data and provided workshops for school and division 
administrators. The research and development work leading to the handbook began in 
the fall of 1998. Much of the work was done in partnership with the GBPC in Marion, 
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Virginia, which had received a request for assistance from the Scott County School 
Division, a local education agency (LEA). AEL collaborated with the Marion GBPC in 
conducting the intensive development site work in the LEA. As the work progressed, 
direct technical assistance was shifted from AEL to the GBPC as a project goal has been 
to build GBPC and LEA capacity to use data for improvement of planning and 
instruction. 

In April 1999, the Marion GBPC and AEL, with GBPC staff taking the lead, 
conducted a workshop on strategies for improving instruction for school and school 
division administrators. Workshop participants consisted of the division superintendent, 
key central office administrators, and building principals. Sixteen workshop attendees 
completed a post workshop evaluation form. The results from this rating form are 
reported below. 

Workshop participants were asked what position(s) they held in the schools or 
central office. Many respondents checked more than one response so all participants are 
listed in Table 1 with the positions they reported to hold. Fourteen of the participants 
were administrators at the schools, and two participants were from the central division 
office. 



Table 1: Positions held by participants at April 1999 workshop in Scott County, 

Virginia 



Partic- 

ipant 


Central 

Office 


K-3 

Administrator 


Gr. 4-5 

Administrator 


Gr. 6-8 

Administrator 


Gr. 8-12 
Administrator 


Other 

Position 


i 




✓ 


✓ 






K-6 

Admin- 

istrator 


2 




✓ 


✓ 








3 








✓ 


✓ 




4 












Not 

specified 


5 












Not 

specified 


6 




✓ 


✓ 


✓ 






7 




✓ 


✓ 


✓ 






8 




✓ 


✓ 


✓ 






9 










✓ 




TO 










✓ 




n 










✓ 




n 






✓ 


✓ 






D 




✓ 








K-4 

Principal 


R 




✓ 










13 


✓ 












re 


✓ 
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Overall Rating of Workshop. Workshop participants were asked to rate the 
overall quality of the workshop on a 5-point scale, from l=poor to 5=outstanding. The 
average score was 4.08, or “above average.” Ratings ranged from “average” (n=2) to 
“above average” (n=8) to “outstanding” (n=3). Three participants did not provide a 
score. 



Usefulness of Information. On the same 5-point scale, respondents rated four 
aspects of the workshop session. The results are presented in Table 2. Reactions to the 
workshop were positive. The average rating given to the workshop aspects was an 
“above average” score. The most variation in responses was found for the planning 
activity. While half of the respondents ranked that activity as “above average,” 25% 
rated it as “average,” and 25% gave it an “outstanding” score. 

Usefulness of Presentation Modes and Materials. Participants were asked to 
provide feedback on the presenters and the materials provided at the workshop. 
Respondents marked a continuous line between 0 and 50, with 0 indicating a “not at all” 
score and 50 representing a “very much” score. These scores were multiplied by two to 
provide a percentage between 0 and 100%, similar to common grading systems, in order 
to facilitate interpretation of the scores. Table 3 presents the “grades” given by the 
workshop attendees. The materials provided to the workshop attendees received mixed 
ratings as to their usefulness. The standard deviation indicates that some of the 
participants found the materials to be useful but others did not find them as useful. The 
presenters were considered to be skilled and knowledgeable, and were generally 
responsive and provided clear explanations. 

Table 2: Descriptive statistics on the items assessing the usefulness of workshop 

session elements 



Workshop Session Element 


N 

Valid Missing 


Mean 


Median 


Mode 


Standard 

Deviation 


Information/strategies regarding 
curriculum and instruction 


16 


0 


4.13 


4.00 


4 


.62 


Information/strategies regarding 
“observation” and other aspects of school 
organization and structure that enhance 
learning 


16 


0 


4.13 


4.00 


4 


.62 


Intormation/Strategies regarding 
accessing information resources, e.g., 
using technology 


16 


0 


4.00 


4.00 


4 


.63 


Planning activity 


16 


0 


4.00 


4.00 


4 


.73 
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Table 3: Descriptive statistics on the items assessing the presentation of the 

workshop 
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Workshop Presentation Element 


N 

Valid Missing 


Mean 

“Grade” 


Standard 

Deviation 


Responsiveness ot the presenters to your needs 
during the session 


15 


1 


84% 


9.9% 


Extent ot presenters’ skills and knowledge 


16 


0 


87% 


9.4% 


Clarity ot presenters’ explanations 


16 


0 


84% 


9.8% 


Usefulness ot materials provided 


16 


0 


80% 


14.3% 



Participants’ Comments and Suggestions. Workshop attendees were asked three 
questions to which they wrote responses. The answers given to these questions are 
organized by how each respondent rated the workshop overall. Please refer to Tables 4-6 
for these written responses. 

In sum, the workshop in Scott County was generally well received by the school 
and division administrators who attended. The presenters were acknowledged as skillful 
and as having a lot of valuable information to share with school personnel. Some of the 
participants reported that the workshop felt too rushed, and would have liked to have had 
more time. There was an acceptance among administrators that the Virginia SOLs were 
there to stay and that they needed to prepare themselves and their staff to increase the 
quality of teaching in order for students to achieve passing scores. 



Table 4: Participants’ responses to the prompt, “What I really liked and why.” 



Participants’ overall rating of the 
workshop 


What participants really liked and why 


(Blank; Did not answer question) 


+ Review of lesson model. 


3=Average 


+ I like having a source for obtaining information (the websites 
provided). The Powerpoint presentation should be used by all 
staffers. 


4=Above Average 


+ Hunter’s Lesson / Time management. 

^ The fact that it dealt with the school/classrooms. 
+ Focusing everyone on objective (SOLs). 

+ Review of PAT 


5=Outstanding 


+ The Best Practice Center’s people’s attitude (positive) in 
assisting us in Scott County and our quest to achieve passing 
scores on the SOL testing. 

+ This session was a good “break” from the earlier ones in that it 
didn’t deal with SOLs directly or scores — but did deal with 
leadership, evaluations and things we as principals deal directly 
with in helping teachers. 

+ Presenters are very “down to earth” and present useful materials. 
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Table 5: Participants’ responses to the prompt, “What I would recommend doing 

differently and why” 



Participants’ overall rating of the 
workshop 


What participants would recommend doing differently and why. 


3=Average 


+ Do a workshop on Steven Covey. I enjoy him. Provide data on 
that “wealth of knowledge.” Show some actual databases. 


4=Above Average 


«No comments» 


5 Outstanding 


+ Longer session — it seemed all presenters were rushed. I would 

have liked to learn more. 

+ I think time was used very effectively — somewhat rushed in 
presentation; however, still very effective because of 
personalities of presenters! 



Table 6: Other comments/suggestions 



Participants’ overall rating of the 
workshop 


Other comments/suggestions 


3=Average 


+ Steve Sage spoke about students’ success before moving on to 
another objective. This issue can be debated. Do I need success 
for all students? What level of mastery are you talking about? 
At some point educators must learn to be effective planners. 
Allocate sufficient time for each SOL and move on. There can 
be levels of success for most. 


4=Above Average 


«No comments» 


^Outstanding 


+ Doris Redfield’s information has been very useful. 



As noted earlier, AEL and staff from the GBPC in Marion, Virginia, worked with 
the Scott County central office staff, school board, and principals to develop and provide 
training on using SOL data at the district, school, and classroom levels between 
December 1998 and April 1999. A result of this effort was a draft handbook for use by 
GBPC staff, which was to be critiqued by those receiving April 1999 training on the 
handbook content. Although feedback on the handbook has not been forthcoming from 
GBPC staff, in part due to changes in GBPC leadership, the AEL-IHE (Institutions of 
Higher Education) Co-Venture awarded an internship to a University of Tennessee 
doctoral student to update the handbook so that it now contains three years of data. The 
data sets and training materials are currently being produced. 



AEL intends to collaborate with the Eisenhower Regional Consortium for 
Mathematics and Science Education to revise the handbook so that it is more directly 
useful to school and classroom practitioners. In addition, it is hoped that the revised 
product will be applicable beyond Virginia. 

Algebra I Resources for Virginia Teachers 

As indicated by the 60% failing rate on the first administration of the Algebra I 
SOL test in the spring of 1998, many students in Virginia have had difficulty mastering 
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Algebra I concepts. Prealgebra and Algebra teachers must focus their instruction to 
address the 19 Algebra I SOLs. Although various resources have been available to 
Virginia teachers as they undertake this new challenge, many continued to request user- 
friendly information about the state standards, specific examples of appropriate 
instructional methods and assessments, and further detail about the Algebra I concepts 
covered in the SOLs. 

AEL and the Harrisonburg GBPC collaborated during 1999-2000 to develop a 
resource compendium for Algebra I teachers that would compile existing example 
lessons and create others to address teachers’ needs. AEL conducted a study group of 
Algebra I teachers to field test and refine activities correlated to the SOLs. Participants 
were solicited with the assistance of the Harrisonburg GBPC staff, and were offered 
advance use of the compendium, networking opportunities with other Algebra teachers, a 
small stipend of $200, mileage reimbursement for meeting participation, 
acknowledgment in the final product, and re-certification credit as locally approved. In 
return, study group participants were asked to participate in an orientation meeting, assist 
in the development of the compendium, field test suggested activities, provide written 
feedback on the materials used and the guide itself, and participate in three meetings to 
share experiences with others and to advise guide developers. 

A total of four Algebra I teachers participated in the study group. Although six 
were originally recruited, two were unable to participate fully. The study group 
facilitator, an AEL staff member, reported that although a larger sample of participants 
might have added richness and depth to evaluations of the compendium, four appeared to 
be sufficient. 

The study group convened three times at the Valley Vocational-Technical Center 
in Fishersville, Virginia. An initial meeting was held in August 1999 with the study group 
members and AEL, GBPC, and VDOE staff to introduce the project. 

The second meeting was held February 29, 2000. At this gathering, participants 
discussed some of their evaluations of various activities in the compendium, shared ideas 
about activities they had developed for inclusion in the resource, and received some 
professional development from a consultant. In addition, as the facilities included a 
computer lab, participants gained experience submitting their assessments of 
compendium activities using an on-line lesson plan evaluation tool. 

Participant reaction to the February 29 meeting of the Algebra resource guide 
study group is presented in the second column of Table 7. Respondents were asked to rate 
their level of agreement with 30 statements, using a 5-point scale, with 1 indicating 
strong disagreement and 5 indicating strong agreement. All four participants completed 
and returned the evaluation form. 

Participants tended to agree strongly with most positively worded assessments of 
the meeting. Mean ratings for these items ranged from 3.50 (indicating moderate 
agreement that facilities at the study group meeting site were adequate, and moderate 
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agreement that participants found it easy to find time to contribute to the group’s 
listserv), to 5.00 (indicating a very high level of agreement that the study group listens to 
individuals’ contributions and that the facilitators kept the group on task). 

With mean ratings of 1.50, the negatively worded items received little agreement 
from respondents. Participants reported, for instance, that the study group’s listserv and 
on-line lesson plan evaluation form were not confusing. 

The February 29 meeting of the Algebra I resource guide study group appears to 
have been quite successful. Participants believed they were being skillfully facilitated 
through a useful process ultimately leading to the development of a needed user-friendly 
and comprehensive resource compendium with potential for impacting student 
achievement in Virginia. 

A third and final working meeting was held June 1, 2000. During this event, 
participants again discussed their appraisals of compendium activities, shared ideas, but 
did not receive any professional development as the consultant was unavailable. The 
AEL facilitator was present. 

Participant reaction to the June 1 meeting of the study group is shown in column 3 
of Table 7. All four participants completed and returned the evaluation form. As before, 
they were asked to rate their agreement with 30 statements using a 5 -point scale for their 
responses. Because the number of participants is so small, mean ratings should be 
interpreted with caution. 

Ratings for positively worded items were again high. Although participants 
reported less agreement than they had in the previous evaluation that it was easy for them 
to find time to post to the study group’s listserv and complete the on-line lesson plan 
evaluation forms, with means of 2.50, they also disagreed that both electronic forums 
were confusing to use (1.00). 

No items received mean ratings of 5.00, but many closely approximated such a 
high rating with means of 4.75. Respondents strongly agreed that the study group was 
representative of Virginia Algebra teachers, that there were ample opportunities for 
discussion and questions at the group’s meetings, that the ultimate product would be 
useful and make an impact on student learning, and that the group was committed to its 
work. 



To conclude, the June 1, 2000, study group meeting appears to have been very 
well received by participants. Finding time for contributions to the listserv and the on-line 
lesson plan evaluation may have been of some concern to respondents, but the study 
group process, facilitators, participants, and goals were viewed positively by those 
involved. 
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Table 7: Mean ratings of Algebra Resource Guide study group meetings, 

N=4 



Item 


February 
29, 2000 
Mean 


June 1, 
2000 
Mean 


1 . Study group meetings have been heid at convenient locations. 


4.50 


430 


2. The people m the study group are representative ot the variety ot Algebra 
teachers in Virginia. 


430 


4.75 


3. 1 understand the purpose(s) ot the study group. 


475 


4.50 


4. The study group listens to what I have to say. 


530 


4.50 


5. The facilities at the study group meeting site(s) ls/are adequate. 


130 


430 


6. My opinion is important to this Algebra resource guide development 
process. 


4.75 


4.50 


7. There were ample opportunities for questions and discussion at study group 
meetings. 


430 


4.75 


8. The professional development activities during the study group meetings 
have been informative. 


4.75 


4.25 


9. The professional development activities durmg the study group meetings 
have offered me the opportunity to develop skills that are relevant to my needs. 


4.50 


430 


10. The facilitators of the study group try to make sure everyone’s opinions 
are heard. 


4.75 


4.50 


1 1 . The facilitators of the study group keep the group on task. 


5.00 = 


4.75 


12. 1 believe that the guide we are developing will help to increase Algebra 
scores on the SOL exams. 


4.75 


4.75 


13. Lessons from the resource guide will help Virginia students better 
understand Algebra. 


4.25 


430 


14. lam satisfied with the direction in which the study group is going. 


4.50 


4.75 


15. The activities from the resource guide address a variety of learning styles 
and developmental needs. 


4.50 


4.25 


16. The people involved in the study group are deeply committed to doing 
their part to make the resource guide better. 


430 


4.75 


17. The facilitators ot this study group really know how to collaboratively 
develop instructional materials. 


430 


430 


18. Being involved in the study group has been a very valuable experience for 
me. 


430 


4.75 


19. I believe that the study group has the capability of helping to produce an 
Algebra resource guide that will benefit Algebra students in Virginia. 


430 


4.75 


20. The activities withm the resource guide were useful to me. 


430 


430 


21. The activities within the resource guide will be useful to Algebra teachers 
in Virginia. 


4.75 


4.75 


22. The advance communications that 1 received m the mail from AEL and/or 
the Governor’s Best Practice (BPC) were useful to me. 


4.75 


4.75 


23. 1 found the study group’s hstserv to be confusing to use. 


1.50 


T30 


24. I found the online lesson plan evaluation form confusing to use. 


1.50 


1.00 


25. 1 was easy for me to find enough time to post on the listserv. 


130 


230 


26. It was easy for me to find enough time to complete the online lesson plan 
evaluation forms. 


430 


230 


27. It was easy for me to find a computer with e-mail access so that 1 could 
post to the listserv. 


4.50 


4.75 


28. It was easy for me to find a computer with Internet (World Wide Web) 
access so that I could complete the online lesson plan evaluation forms. 


4.50 


4.75 


"~29. Each section of the resource guide includes all the information it needs to 
include (i.e., it is comprehensive). 


3.75 


4.25 


Jo. The resource guide is easy to use. 


435 


430 




11 



At the June 1, 2000, meeting, study group members were also asked to provide 
feedback about the overall experience of participation. A questionnaire with seven open- 
ended questions solicited participants’ evaluations of the study group process, challenges 
to participation, future use of the compendium, and feedback about the collaboration 
between AEL, VDOE, and the GBPCs. 

All four study group members reported that they had gained useful information or 
resources as a result of their participation. One additionally noted that he or she had 
gained knowledge about technology. Asked about the challenges associated with their 
involvement in the study group, three participants said that time had been an issue: Two 
noted that the time needed to complete field tests of all the compendium activities had 
presented difficulty, and one reported that scheduling meetings had been challenging 
although completed successfully. Another participant wrote that competing demands 
deflected his or her attention from study group obligations. 

Two respondents suggested that the challenges mentioned above might have been 
minimized had the group met more frequently. Another two offered praise for the support 
that had been offered during the project to address some of the difficulties: One liked the 
study group listserv, and one appreciated “how easy it was to get answers to my questions 
or concerns.” 

All participants indicated that they intended to use the final version of the 
resource guide in their classrooms. As one put it, “I plan to further use the lessons and 
also the sponge activities. I think that the presented resources will also be a big help to 
me and others in the future.” 

Asked who should receive the completed compendium, all four replied that at 
least one copy of the guide should be distributed to every school, so that every Algebra I 
teacher had access to it. Respondents had differing opinions about the need for 
professional development about the guide for those receiving it. Two reported that 
training on the compendium was not necessary. “I really do think that this document can 
stand by itself!!” wrote one participant. The remaining two respondents suggested that 
some professional development might be helpful. 

Participants were asked to provide feedback about the collaboration between 
AEL, VDOE, and the GBPC. Two respondents indicated that the project would have 
been strengthened with more representation and input from VDOE staff, and another 
suggested that “making sure that all teachers are aware of what’s available and have 
access to those materials” would be a wise strategy. One study group member reported 
that a strength of the collaboration had been the “online/e-mail networking which kept us 
all connected even though we were all over [the state].” A weakness of the collaboration, 
wrote one respondent, was that “we were included on [sic] ‘brainstorming’ that I’m not 
sure we were needed for.” 
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In reply to the final open-ended question soliciting any additional comments, 
questions, or suggestions, all four respondents expressed appreciation for the study group 
experience. For instance, one participant reported that the facilitator “did an outstanding 
job responding to e-mail, coordinating meetings, and adjusting to unexpected events.” 
Suggestions included developing similar resource guides for Geometry and Algebra II 
teachers, “alignment with the CASlOs rather than the 83s,” creating overhead 
transparencies to coordinate with sponge activities, and distributing the final 
compendium to all Algebra I teachers. 

Participant feedback, then, suggests that the study group was well-coordinated 
and useful as professional development to those involved. Although time and competing 
demands were challenges participants faced, these were not insurmountable. Teachers 
intended to use the final compendium in their own classrooms and hoped that it would be 
widely distributed. The study group listserv was seen as a strength of the experience, 
assisting with communication and coordination of work. Some respondents would have 
liked more participation from VDOE staff during the process. Overall, though, 
participants were pleased with their experience. 

In terms of the study group product, the Algebra consultant who had provided 
professional development to the participants was also engaged to analyze the on-line 
lesson plan evaluation forms to complete revisions of the compendium. Unfortunately, 
due to changes in the consultant’s other professional obligations, such analysis has yet to 
be completed. AEL staff are currently considering hiring another Algebra consultant to 
bring the project to closure. 

CSRD External Facilitator Training for GBPC Staff 

An abbreviated version of AEL’s Academy of External Facilitators was provided 
to GBPC and the VDOE Office of Compensatory Programs staff November 1-2, 1999, in 
Richmond, Virginia. A total of 22 individuals participated in this training, 15 of whom 
were staff of GBPCs. 

Presented as an orientation to external facilitation, the event’s goals were to (1) 
build an awareness of the roles and responsibilities of an external facilitator of schools 
undertaking reform funded through the Comprehensive School Reform Demonstration 
(CSRD) program; and (2) prepare an engagement plan for helping low-performing 
schools build capacity for improvement. In addition, AEL CSRD staff articulated daily 
objectives to be achieved during the training. Objectives for the session on November 1 
were to 



• examine school reform and the CSRD program from the national perspective 

• introduce the role of the external facilitator in comprehensive school reform 

• increase the capacity to analyze and understand school culture 

• understand the impact of key influences on a school undergoing change including 
school culture, state education contexts, externally developed models, and the 
nine CSRD components. 
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Objectives for November 2 were to 

• make inferences as to why schools demonstrate certain strengths and weaknesses 

• stimulate reflection and discussion about strategies leading to action for change 

• develop and refine expertise to work more effectively with groups 

• develop an engagement plan for working with individual schools. 

The training was designed and presented by AEL CSRD staff in conjunction with 
several consultants who served as members of the Academy for External Facilitators 
design team. It should be noted that external facilitation with regard to CSRD is defined 
somewhat differently than the external facilitation function described in the Introduction 
of this report. 

Because it was not deemed cost-effective to evaluate the training in a 
comprehensive manner, assessment of the workshop was conducted via the 
administration of a final evaluation questionnaire. Only 12 of the 22 participants 
completed and returned this instrument, representing a return rate of 55%. It is unclear 
why so few forms were returned. 

Section A of the evaluation form asked respondents to reply to items related to 
various logistical aspects of the institute using a 5-point scale, with 1 indicating a 
response of “never or not at all” and 5 indicating a response of “always or very.” Section 
B asked respondents to rate the extent to which various institute objectives were met 
using a scale identical to that in Section A. Tables 8 and 9 present data from these two 
sections of the evaluation form, including mean ratings and the percent of those agreeing 
or strongly agreeing with statements. Finally, Section C posed five open-ended questions 
concerning the helpfulness of the event, suggestions for improvement, further assistance 
AEL might offer, and any remaining concerns. 

Overall, participants rated logistical characteristics of the orientation fairly high 
(see Table 8). Receiving the highest mean rating was the extent to which AEL staff and 
presenters were responsive to attendees’ requests for assistance and service (4.46, SD 
.69). Staff were skilled (4.33, SD .78) and knowledgeable (4.33, SD .98), according to 
those who replied to the instrument. Respondents were somewhat less satisfied, however, 
with the format of the orientation, with a mean rating of 3.50 (SD 1.24); only 50% of 
respondents reported being satisfied with this aspect of the event. The large standard 
deviation also suggests that there was some disagreement among respondents on this 
issue. 



Ratings on Section B items suggest that respondents felt the extent to which the 
orientation achieved goals and objectives was adequate, although not outstanding (see 
Table 9). The highest mean rating of 4.42 (SD .67) was given for the extent to which 
respondents learned from other participants. The degree to which respondents’ 
knowledge of strategies leading to action for change increased received the lowest mean 
rating of 3.50 (SD 1.00). Also of note is the large standard deviation (1.51) for responses 
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to the item concerning the adequacy of technical assistance and support received for the 
development of engagement plans. The mean for this item, 3.58, indicates that such 
assistance could have been strengthened for some respondents. 

Table 8: Mean ratings on external facilitation orientation final evaluation form, 

section A 



Item 


N 


Percent 

Agree 


Mean 


SD 


1. Did AEL carry out planned activities at the times scheduled? 


12 


75% 


4.17 


.83 


2. How responsive were AEL statt and/or presenters to your 
requests for service and/or assistance during the orientation? 


12 


9U% 


T45 


.69 


3. How knowledgeable were AEL staff and/or presenters? 


12 


83% 


4.33 


.98 


4. How skilled were AEL statt and/or presenters in completing 
their tasks? 


12 


83% 


4.33 


.78 


5. How well were your information needs met at this orientation? 


12 


58% 


3.75 


1.36 


6. How useful were the materials provided to you during this 
orientation? 


12 


57% 


3.92 


1.17 


7. How satisfied were you with the format of this orientation? 


12 


50% i 


3.50 


1724 


8. How useful do you anticipate this orientation will be to your 
work? 


12 


67% 


4.08 


T3J8 



Table 9: Mean ratings on external facilitation orientation 
final evaluation form, section B 



Item 


N 


Percent 

Agree 


Mean 


SD 


1. To what extent did you acquire an understanding of the roles 
and responsibilities of external facilitators? 


12 


58% 


3.75 


1729 


2. To what extent did the orientation help you increase your 
capacity to analyze and understand school culture? 


12 


57% 


TDD 


F705 


3. To what extent did you knowledge of the CSRD program 
increase? 


12 


58% 


3.75 


1.36 


4. To what extent did you receive adequate technical assistance 
and support for developing your engagement plan? 


12 


57% 


3.58 


1.51 


5. To what extent did you learn from other participants? 


12 


92% 


T42 


.67 


6. To what extent did you develop/refine your expertise to work 
more effectively with groups? 


12 


75% 


T92 


.90 


7. To what extent did your knowledge of strategies leading to 
action for change increase? 


12 


50% 


3.50 


TDD 


8. To what extent did your understanding of the impact of school 
culture on reform initiatives improve? 


12 


58% 


3.75 


.75 


9. To what extent did your skills for analyzmg school culture 
improve? 


12 


57% 


3.83 


.94 


10. To what extent did your understanding of your own problem 
solving approaches improve? 


12 


58% 


3775 


.97 


1 1 . To what extent did your understanding of others’ problem 
solving approaches improve? 


12 


58% 


3.75 


.75 



Asked in Section C of the evaluation form what had been the most helpful part of 
the orientation in terms of helping respondents work in low-performing schools, nine of 
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the 12 total respondents replied. Two offered responses with multiple themes. Two 
respondents each reported that the sharing of ideas and the materials had been the most 
useful components. The remaining replies were idiosyncratic. Of use to these respondents 
were descriptions of the external facilitator role, the Kirton Adaption-Innovation 
Inventory, the group synergy exercise, specific steps described for analyzing school 
plans, and the orientation’s interactive format. One respondent gave a more critical 
answer: “If it weren’t for Ben Perez [presenter], I wouldn’t have gotten anything from 
these two days.” 

Four respondents answered the question about what concerns they still had about 
their work in low-performing schools. One reported having no concerns, and one was 
concerned about finding sufficient time to work as an external facilitator while also 
maintaining her other duties. Two respondents had concerns about their new role; one 
asked, “Can I really be of assistance, or will I be in the way?” The second wondered 
“how to encourage schools to cover gap areas.” 

Asked what assistance they might want AEL to provide as external facilitators 
began their work in low-performing schools, six of the 12 respondents replied. All six 
requested in some manner that AEL keep in contact with participants, sending updates 
and helpful tips. 

Ten respondents replied when asked in what ways the orientation could be 
improved. Three of these gave responses with multiple themes. Three comments each 
suggested that the content of the orientation was too basic given the knowledge and 
experience of the participants and that attendees would have preferred more active 
engagement. Two respondents recommended that the storytelling portion of the 
orientation be limited or eliminated. One suggested that the working lunch be cut, and 
another requested more breaks. The presentation style and delivery of some orientation 
presenters concerned a respondent, who wrote that there had been “too much bird 
walking, rambling, elaboration.” Two remaining respondents reported that they had no 
recommendations. 

Nine participants responded to the final query, “Is there anything else you’d like 
to tell us about your experiences at the orientation to external facilitation?” Five 
comments simply offered praise or thanks. For example, one respondent wrote that the 
orientation was a “very good experience,” and another said, “All presenters made me feel 
comfortable learning.” Four comments, however, were more critical. Three of these 
suggested that the content of the orientation had limitations: Two respondents thought 
their knowledge levels had been underestimated, and a third reported that the orientation 
focused too much on process at the expense of content. Another respondent felt the pace 
was too slow and suggested more carefiil editing of PowerPoint presentations. 

In sum, the orientation was fairly well received by participants. However, 
evaluation respondents had a few concerns. Already versed in education reform research, 
some reported that the content of the training was too basic. Others requested more 
engaging activities and less storytelling. Respondents reported that the orientation’s goals 




22 



16 



and objectives had been achieved, although perhaps not to the very highest degree. 
Hence, the orientation was relatively successful, and might have been more so with a few 
modifications. 
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Conclusions 

Several conclusions may be drawn based on the data collected. 

It appears that the AEL Resident Director participated in GBPC meetings and 
facilitated assistance to the centers based on their stated needs. In addition to aiding in the 
facilitation of strategic planning meetings, AEL staff also collaborated with GBPC staff 
to offer assessment training, develop an assessment handbook for center staff, facilitate a 
study group for the refinement of a compendium of resources and strategies to assist 
Algebra I teachers to accommodate the SOLs in instruction, and provide training in 
external facilitation. In addition, the two proposed products, the assessment handbook 
and the Algebra I resource compendium, are advancing toward completion. 

Participants in processes and meetings co-developed and co-facilitated by AEL 
tended to be somewhat to highly pleased with these services. The Algebra I study group 
was particularly satisfactory in this regard. The CSRD training for external facilitators, 
however, was less well-received. 

The content presented at the CSRD training for external facilitators may have 
been too basic and introductory given the levels of knowledge and experience held by 
state department and GBPC staff. 

It may be that as AEL works in partnership with others to build capacity for state- 
directed technical assistance for education improvement initiatives, greater flexibility of 
each partner is required to work directly with schools and districts. The success of AEL’s 
work with the Algebra I study group in which AEL staff were able to work directly with 
individual teachers through collaborative efforts and assistance of the GBPC in the region 
suggests this. Obviously, the effectiveness with which AEL carried out its work with the 
GBPCs may have been influenced by the political realities and uncertainties that they 
confronted as new creations of the state department of education. As a viable partner 
seeking to serve as a catalyst for stimulating opportunities for schools and divisions to 
access needed “external” technical assistance for school improvement initiatives, AEL 
will need to continuously build and nurture short- and long-term collaborative 
relationships when the development of products involves several groups who may have 
competing priorities. 
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Appendix: 

Completed Evaluation Standards Checklist 



Checklist for Applying the Standards 

To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 


The Standard was 
addressed 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U1 


Stakeholder Identification 


X 








U2 


Evaluator Credibility 


X 








U3 


Information Scope and Selection 




X 






U4 


Values Identification 


X 








U5 


Report Clarity 


X 








U6 


Report Timeliness and Dissemination 


X 








U7 


Evaluation Impact 


X 








FI 


Practical Procedures 


X 








F2 


Political Viability 


X 








F3 


Cost Effectiveness 


X 








PI 


Service Orientation 


X 








P2 


Formal Agreements 


X 








P3 


Rights of Human Subjects 


X 








P4 


Human Interactions 


X 








P5 


Complete and Fair Assessment 


X 








P6 


Disclosure of Findings 


X 








P7 


Conflict of Interest 


X 








P8 


Fiscal Responsibility 


X 








A1 


Program Documentation 


X 








A2 


Context Analysis 


X 








A3 


Described Purposes and Procedures 


X 








A4 


Defensible Information Sources 


X 








A5 


Valid Information 


X 








A6 


Reliable Information 


X 








A7 


Systematic Information 


X 








A8 


Analysis of Quantitative Information 


X 








A9 


Analysis of Qualitative Information 


X 








A10 


Justified Conclusions 


X 








All 


Impartial Reporting 


X 








A12 


Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one). 



request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

x evaluation report 

other: 



Name 



Caitlin Howley-Rowe 



Date 






Position or Title 



(signature) 
Research Associate 



AEL, Inc. 



Agency 

Adclress 1031 Quarrier Street Charleston, WV 253CLL 



Relation to Document 



Co-author 



(e.g., author of document, evaluation team leader, external auditor, internal auditor) 



11/29/Qa 
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